fix(temporal): guard int64 overflow in DATE/TIMESTAMP decomposition - #386
Conversation
The DATE/TIMESTAMP → microseconds conversion shared by the temporal
extract and truncate paths overflowed int64 on extreme inputs (UBSan):
* DATE is int32 days, so an extreme value × µs-per-day overflowed —
(dd (as 'DATE 2147483647)) tripped `raw * 86400000000`.
* The ns→µs floor for TIMESTAMP negated the input, so a value within
999 of INT64_MIN overflowed `(-raw) + 999`.
* The DAG date_trunc YEAR/MONTH arms re-multiply days_from_civil(...) —
a day count floored down to the period start, up to a year beyond the
input — by µs-per-day, so a DATE that cleared the µs bound still
overflowed (d.year of (as 'DATE -106751991)).
Both conversions appear in all four decomposition kernels: the standalone
ray_temporal_extract / ray_temporal_truncate and the DAG exec_extract /
exec_date_trunc morsel kernels.
Do the TIMESTAMP ns→µs floor with truncate-then-adjust so it never
negates (overflow-free, exact at INT64_MIN). A DATE so extreme its µs
value is not representable decodes to a null instead of reading overflow
garbage, consistent with how these kernels already treat a null input.
The truncate DATE bound is the int64-NANOSECOND representable day range
(not just the µs one) so the YEAR/MONTH re-multiply cannot overflow, and
truncate additionally nulls a result whose bucketed µs would overflow the
ns output.
Adds temporal/extract_trunc_overflow.rfl covering the standalone kernels
(yyyy/dd/mm/hh and (date …)) and the DAG kernels (dotted col.field),
including the year/month re-multiply path, large-magnitude negative DATE,
and minimum-edge TIMESTAMP.
bfa94e1 to
ae215c5
Compare
|
Addressed the Blocking (1) — incomplete truncate guard. Correct: the (2) — vacuous assertion. Right, (3) — untested standalone truncate / overstated header. Added (4) — conservative low-edge guard. Kept the unconditional one-bucket headroom (avoids a branch in the hot kernel) and documented that it rounds an immaterial ~1-bucket band at the extreme low edge (~292 millennia before 2000) to null. Full |
singaraiona
left a comment
There was a problem hiding this comment.
Found one correctness issue at the lower timestamp boundary. The overflow fixes otherwise look sound, CI is green, and the full local ASan/UBSan suite passed (3664/3664).
| static inline bool rte_trunc_elem(int8_t t, int64_t raw, int64_t bucket, int64_t* out_ns) { | ||
| int64_t us; | ||
| if (!rte_to_us_ck(t, raw, &us)) return false; | ||
| if (us > INT64_MAX / 1000LL || us < INT64_MIN / 1000LL + bucket) return false; |
There was a problem hiding this comment.
[P2] This unconditional bucket headroom rejects valid timestamps before checking the actual truncated result. For example, (date (as 'TIMESTAMP -9223286400000000000)) returns 0Np, although that value is already the representable day boundary 1707.09.23D00:00:00.000000000. The equivalent DAG path (select ts.date) returns that timestamp, so the two public paths disagree. Please compute the bucketed value with overflow-safe arithmetic, range-check the actual result, and add this boundary as a regression.
rte_trunc_elem rejected a truncation whenever the input fell within one bucket of the low int64-ns edge, even when the floored result was still representable — so (date (as 'TIMESTAMP -9223286400000000000)) returned 0Np though its day boundary 1707.09.23 is a valid TIMESTAMP that the DAG path (select ts.date) returned, leaving the two public truncate paths disagreeing (PR RayforceDB#386 review). Floor with overflow-safe arithmetic (truncate toward zero, then guard the single toward-minus-infinity bucket subtraction) and range-check the actual bucketed result against the ns domain, mirroring exec_date_trunc. Adds the low-boundary case — standalone and DAG — as a regression.
|
Addressed the review in
Added that boundary — standalone |
* v2.4.0 (#327) * feat(query): support live inserts into parted tables Add immutable live-tail growth with explicit partition keys, shared FILE-domain symbol handling, atomic symbol rebinding, adversarial coverage, documentation, and a runnable rollover example. * fix(core): restore total-core -c semantics * fix(parse) Fix nonstring if not defined * fix(store): surface FlushFileBuffers failure in journal SYNC mode (#335) In RAY_JOURNAL_SYNC mode ray_journal_write_bytes checks fsync's return on POSIX and fails the write with RAY_ERR_IO, but the Windows branch ignored FlushFileBuffers' return. A failed flush there was silently swallowed, so SYNC mode reported success while the data may not have reached disk — dropping the durability guarantee the mode exists to provide. Check FlushFileBuffers (0 = failure) and return RAY_ERR_IO, mirroring the POSIX path. Windows-only branch (not built on the Linux/macOS CI matrix), so it is verified by inspection against the adjacent fsync check; the failure path is not unit-testable, like the existing POSIX one. Co-authored-by: Evgen Byelozorov <belowzeroff@gmail.com> * fix(hnsw): reject build dims whose vector count overflows size_t (#333) ray_hnsw_build sized the copied vector block as n_nodes * dim * sizeof(float) with no overflow check. Dimensions whose product wraps size_t under-allocate the copy while the memcpy — and every later distance read (vectors + id*dim) — run past the buffer. Guard the product before any allocation, mirroring the per-layer neighbor guard in the loader, and reject overflowing dimensions. This hardens the public C API boundary; the in-tree (hnsw-build ...) path sizes vectors from an in-memory list and cannot reach the overflow, so it is defense-in-depth. Add a regression test driving an overflowing n_nodes/dim pair; with the guard removed it faults under ASan (stack-buffer-overflow at the copy). Co-authored-by: Evgen Byelozorov <belowzeroff@gmail.com> Co-authored-by: Anton Kundenko <singaraiona@gmail.com> * fix(store): read full link sidecar to avoid wrong-symbol truncation (#334) try_load_link_sidecar read the target table's sym name into a fixed 256-byte buffer (fread of 255 bytes). A name longer than 255 bytes was silently truncated, so ray_sym_intern interned a DIFFERENT symbol and the loaded column linked to the wrong table — silent data corruption on a save/load round-trip. The writer already emits the full, untruncated name. Read the whole sidecar into a buffer sized to the file (capped at 1 MiB to bound a corrupt/oversized file), and reject a short read (fread returning fewer bytes than the file size — an I/O error or a race-truncated sidecar) so a partial name can't be interned as a different symbol either. Add a regression test that links through a 300-byte target name and asserts the loaded link_target matches; it fails without the fix. Co-authored-by: Evgen Byelozorov <belowzeroff@gmail.com> Co-authored-by: Anton Kundenko <singaraiona@gmail.com> * fix(hnsw): reject index files whose vector count overflows size_t (#332) hnsw_load_impl read n_nodes and dim straight from the file header and sized the vectors allocation as n_nodes * dim * sizeof(float) with no overflow check. A crafted header could make that product wrap size_t, so ray_sys_alloc under-allocated the buffer while the following fread still read the full (large) element count and wrote past the allocation — a heap-overflow write driven by an untrusted index file. Factor the check into ray_hnsw_vec_size_valid(n_nodes, dim) and reject the header before any allocation, mirroring the per-layer neighbor guard. Add a unit test that drives the helper directly (ordinary dims, non-positive dims, an overflowing pair, and the exact size_t boundary). It is tested at the helper rather than through ray_hnsw_load because an overflow-patched header is refused earlier — the huge node-level read fails first — so a full-load test could not distinguish the guard. Co-authored-by: Evgen Byelozorov <belowzeroff@gmail.com> * fix(docs): remediate F-0001 F-0005 F-0007 Escalate F-0002, F-0003, F-0004, and F-0006 into CF-0001 through CF-0004 after the required corpus census. * fix(docs): remediate CF-0001 * fix(docs): remediate CF-0002 * chore(audit): plan CF-0003 ratification * fix(docs): remediate CF-0003 * feat(docs): redesign website and documentation Rebuild the MkDocs and marketing surfaces around the Rayforce brand, add the live market demo and cloud preview, unify responsive navigation, and eliminate reload layout shifts. * fix(null): avoid f64 null casts to integers (#340) * fix(null): avoid f64 null casts to integers * fix(expr): guard f64 to i64 fallback casts * fix(null): clamp finite f64 narrow casts --------- Co-authored-by: Evgen Byelozorov <belowzeroff@gmail.com> * fix(expr): avoid null truthiness casts in fallback binary ops (#339) * fix(expr): avoid null truthiness casts in fallback binary ops binary_range's fallback OP_AND/OP_OR kernels cast the widened `double` operand straight to `uint8_t`: uint8_t li = (uint8_t)LV_READ(i); `LV_READ` widens integer operands to `double` and yields NaN for float nulls, so this had two defects: - Wrong answers from 8-bit truncation: `(uint8_t)256.0 == 0`, so `256 and 1b` returned false. - Undefined behavior: casting NaN (NULL_F64) or a widened NULL_I64 (-9.2e18) to `uint8_t` is UB per C11 6.3.1.4. UBSan flagged the latter via expr_null/diff_i64_{and,or}_raw and expr_null/diff_f64_andor_chokes. Route AND/OR through two truthiness helpers that compare on the widened double and never cast it back to an integer: - truthy_intish(v, nullv) — false for 0 and for the operand's null sentinel. The fallback reads raw column memory, so a null arrives as the per-type sentinel widened to double (NULL_I16 / NULL_I32 / NULL_I64, with DATE/TIME stored as I32) rather than the NULL_I64 the VM kernel sees; `nullv` is derived per operand from the bound pointer type so I16/I32 nulls read as false, not just I64. - truthy_f64ish(v) — false for 0.0 and NaN (float null). Non-null truthiness is unchanged and null-input positions still agree with the VM kernel (documented "AND/OR with any null operand -> 0" and the fix_null_comparisons post-pass), keeping fallback ≡ fused. Add regression tests pinning fallback ≡ fused for nullable I64, I32 and I16 AND/OR operands (expr_null/diff_i{64,32,16}_{and,or}_raw). * fix(expr): preserve near-sentinel i64 truthiness --------- Co-authored-by: Evgen Byelozorov <belowzeroff@gmail.com> Co-authored-by: Anton Kundenko <singaraiona@gmail.com> * ci: make Rayforce audit PR comments best-effort * ci: publish Rayforce audit comments from trusted workflow * ci: resolve fork PRs for audit commenter * perf: parallelize serial stages around group-by; unify binary-agg null semantics (#341) * wip(group): parallel binary aggregates (pearson/wavg/cov) via DA path Route binary co-moment aggregators through the dense-array (DA) group path instead of the hash scatter path. Adds sum_y/sumsq_y/sumxy co-moment slots to da_accum_t + per-row accumulation + per-worker merge; emit_agg_columns already finalises PEARSON/COV/WAVG/WSUM from the co-moments. Fixes poor multi-thread scaling of by-key binary aggregates (was ~2x, DA path scales ~9-12x like stddev). Verified vs numpy; diff comparator relaxed to 1e-9 combined abs+rel (1e-12 absolute tested bit-identical summation). Includes temporary RAY_GRPPROF phase instrumentation (to remove). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(group): merge binary-agg Sx as double for integer x-columns wavg/pearson accumulate Sx as double even when the x column is integer (e.g. wavg(bsize,bid), bsize=I32). The per-worker merge dispatched on the x-column type -> read the double bits as int64 -> garbage at >1 worker. Force float merge for binary aggs at all 3 sum-merge sites. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(group): merge binary-agg co-moments in parallel da_merge_fn path The parallel slot-range merge (da_merge_fn, taken when n_slots>=1024) merged sumsq but not the binary-aggregate co-moment arrays (sum_y/sumsq_y/sumxy). Multi-key pearson/cov/wavg over >=1024 dense slots produced wrong results at >1 worker. Add the DA_NEED_PAIR merge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(group): remove temporary RAY_NO_V2/RAY_GRPPROF instrumentation The binary-agg DA fix lands on the default path (v2 declines CHAR-keyed binary group-bys -> legacy DA), so the debug env overrides are no longer needed. 3635/3635 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(if): parallel elementwise OP_IF fill; route trivial-branch if to eager exec_if always took the 'selected' lazy-branch path, whose scaffolding (true-count, id-list build, per-branch gather, scatter) is serial over ALL rows — every if-projection ran at single-core speed regardless of -c (100M numeric if: 2.2s at any core count). 1. exec_if_eager: one shared fixed-width elementwise fill, dispatched across the worker pool for len >= 64K (SYM sides warm their runtime-id LUT serially first — sym.c frozen-table rule, mirrors window.c). STR keeps the serial append path. 2. exec_if_selected: bail to eager when both branches are trivial (column scan / scalar const) and eager fills the type combination correctly — the lazy path only pays off when a branch is an expression worth restricting to its passing rows. Mixed numeric/string shapes stay on the selected path (its per-value string conversion). 100M rows local c24: numeric if 1883->310ms, sym if 2012->306ms. dazzle c48 canonical Q22: 2658->1333ms end-to-end. make test 3635/3635. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(filter): parallel bitmap->index build in exec_filter and sel_compact exec_filter ran two sequential 0..nrows sweeps (pass-count and match_idx build) before its parallel gather; sel_compact rebuilt match_idx from the rowsel serially. Both now use the classic 3-phase compaction: parallel per-chunk/per-seg counts, tiny serial prefix, parallel fill at disjoint offsets. Lazy/morsel-backed predicates keep the sequential sweep. 100M rows local c24: 2-col where-select 143->21.7ms (1.4x -> 6.6x scaling); if+where 1040->324ms. make test 3635/3635. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf: parallel where builtin, gather_by_idx, and chunk-task dispatch - ray_where_fn: 3-phase chunk compaction on the pool (was fully serial). - gather_by_idx: fixed-width value gathers dispatched over disjoint output ranges (null-bit propagation stays serial - shared-word bit writes would race). - exec_filter/where chunk phases now use ray_pool_dispatch_n (one task per chunk); ray_pool_dispatch morselizes total_elems by 1024, so passing chunk counts gave only ~2 tasks for 100M rows. 100M rows local c24: where 88->25ms, at-gather 80->31ms, 2-col where-select 138->20ms (6.8x). make test 3635/3635. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(review): harden parallel paths per skeptic review Blockers (DA binary-agg y-column): - eligibility now requires a plain numeric/temporal y; nullable integer/temporal y stays on the HT path (da_accum_row's pair branch has no y-side sentinel machinery - nulls would accumulate as values) - an FP y with HAS_NULLS sets da_any_nullable so nn[] is allocated and the emitter divides by the non-null PAIR count, not the group count Majors: - all new parallel gates require pool->n_workers > 0 (a -c 1 pool exists with 0 workers; ring fill + atomics + rc_sync were pure overhead, and the OP_IF eager reroute lost to the selected path serially - the Q22/Q25 c1 regression) - chunked dispatch_n call sites cap chunks at 1024 = the pool's initial ring capacity, so the ring never grows (dispatch_n clamps and silently DROPS tasks if ring growth fails -> uninitialized prefix entries -> OOB writes) - sel_compact seg fill switched to dispatch_n over seg-chunks (ray_pool_dispatch over segs gave 1 task under 8.4M rows) - gather_by_idx parallel path guarded by ray_parallel_flag == 0 (leaf utility, 35+ call sites; nested dispatch would corrupt the single-producer task ring) Nits: stray time.h include, restored v2-gate comment, RAY_PARALLEL_THRESHOLD symbol in pivot.c. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(group): pair-skip y-side nulls in the legacy HT binary-agg path Unify grouped binary-aggregate (pearson/cov/scov/wsum/wavg) null handling with the scalar reducers, the v2 engine and the DA path: a null on either side of the (x,y) pair now voids the whole pair on the legacy HT route too. - ght_compute_layout: a nullable y-side sets GHT_AF2_Y_NULLABLE and routes the layout to the null-aware accumulators. - accum_from_entry_nullable: pair-skip before nn++/sums. - Entry packing canonicalizes integer nulls so the accumulator can see them: NaN in F64-packed slots (a (double)sentinel cast previously read as a huge finite value — this also fixes nullable-int x beside an FP y), NULL_I64 in int-by-int slots. - Both HT emitters (radix + serial) divided pearson/cov/scov moments by the group row count instead of the accumulated pair count — wrong results whenever a group carried any null; now divide by nn. - The all-null-group guards wrote v=0.0 after ray_vec_set_null, so the common store overwrote the null sentinel — emit NULL_F64 instead. - DA eligibility now also rejects a y shorter than the scan (OP_CONST vector literal would read out of bounds). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JZwH1gpAeLzDX2bxc4jz6s * test(agg): cross-path null coverage for grouped binary aggregates 46 assertions for wsum/wavg/pearson_corr/cov/scov over nullable inputs on all three grouped routes — v2 (plain-scan int key), DA (expression int key), legacy HT (expression key + nullable-int y; F64-packed and int-packed entry lanes) — against independently computed pair-skip truth, for all four x/y type combinations, plus an all-pairs-null group (wsum 0.0, typed nulls for the ratio/moment aggs) on every route. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JZwH1gpAeLzDX2bxc4jz6s * chore(review): shared dispatch-safety gate; single filter threshold - ops/internal.h ray_par_dispatch_ok(): workers + RAY_PARALLEL_THRESHOLD + ray_parallel_flag reentrancy check in one place; applied at exec_filter, sel_compact, exec_if_eager and the where builtin (local copy there — builtins.c cannot include ops/internal.h). - exec_filter: gate and table fallback derive from one row count (fidx_rows); note that pass_count from the parallel count phase is consumed by exec_filter_vec for vector inputs. - group.c: drop the never-read da_ctx_t.agg_pair_mask plumbing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JZwH1gpAeLzDX2bxc4jz6s * chore(par): shared dispatch predicate, ring-cap constant, parallel-path test Follow-ups from the audit's non-blocking notes: - core/pool.h ray_pool_par_dispatch_ok(pool, n, min_elems): the single home for the dispatch-safety predicate (workers + element threshold + ray_parallel_flag reentrancy). The three hand-copies in ops/internal.h, lang/eval.c and ops/builtins.c are gone; all six gates call the shared one. - RAY_POOL_INIT_TASKS in core/pool.h replaces the hardcoded 1024 at the three dispatch_n chunk caps and in ray_pool_create, with a _Static_assert tying it to RAY_POOL_MAX_TASKS — lowering the initial ring capacity can no longer silently desync from the caps that rely on it. - test/rfl/query/parallel_paths_large.rfl: 200k-row coverage of every new pool-parallel branch (where, gather-by-index, exec_filter, sel_compact, OP_IF numeric and SYM fill incl. the serial LUT warm-up) against closed-form expected values. - RAY_F32 dropped from if_type_eager_ok's whitelist (if_fill_range has no F32 case; unreachable today, kept unreachable deliberately). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JZwH1gpAeLzDX2bxc4jz6s --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Anton Kundenko <singaraiona@gmail.com> * fix(aggr): preserve slice nullability in binary groups * fix(expr): avoid f64 null cast in fallback idiv integer output (#344) binary_range's OP_IDIV kernels for narrow integer output (I64/I32/I16/U8) computed `(intN_t)floor(lv/rv)` guarded only by `rv != 0.0`. That guard does not catch a NaN operand (`NaN != 0.0` is true), so a null float input yields `lv/rv == NaN`, `floor(NaN) == NaN`, and the subsequent cast to an integer type is undefined behavior — UBSan: "nan is outside the range of representable values of type 'long long'" at exec/expr_binary_f64_idiv_mod. Route the cast through the ray_cast_f64_to_{i64,i32,i16,u8}_null helpers, which map NaN to the canonical null sentinel (NULL_I64/I32/I16, 0 for the non-nullable U8) and saturate out-of-range finite results. The null post-pass (propagate_nulls_binary) already overwrites these positions, so final values are unchanged — this only removes the UB and yields the correct sentinel in-buffer. Mirrors the already-safe F64-output IDIV arm (ray_f64_fin) and the sibling casts fixed in "avoid f64 null casts to integers"; depends on those helpers. Co-authored-by: Evgen Byelozorov <belowzeroff@gmail.com> * fix(group): avoid f64 null read cast in dense aggs (#343) Co-authored-by: Evgen Byelozorov <belowzeroff@gmail.com> Co-authored-by: Anton Kundenko <singaraiona@gmail.com> * fix(ipc): preserve boxed data list args (#346) * fix(group): avoid f64 null cast in DA reads (#348) * ci: use portable march for fuzz jobs (#347) Co-authored-by: Anton Kundenko <singaraiona@gmail.com> * fix(pivot): preserve generic missing cells as null (#350) * fix(xbar): avoid narrow bucket truncation (#351) * fix(arith): reject float temporal operands (#353) * fix(ops): make nested (LIST) columns usable through a parted view (fixes #355) (#356) * fix(query): preserve temporal arithmetic semantics (#354) * ci: skip audit comments for cancelled runs * fix(query): preserve if temporal branch types (#359) * fix(store): reject duplicate splayed column names (#360) * fix(store): reject duplicate splayed column names * test(ci): harden Ctrl-C PTY synchronization --------- Co-authored-by: Anton <singaraiona@gmail.com> * fix(builtins): reject malformed strings in TIMESTAMP cast (#361) * fix(builtins): reject malformed strings in TIMESTAMP cast (as 'TIMESTAMP str) accepted a range of malformed inputs and silently produced a valid-looking but wrong value — invisible data corruption at the call site. Examples that used to succeed: - "2024-01-02x01:02:03", "2024-01-02abc" -> midnight (time dropped) - "2024-01-02T25:02:03" -> rolled into the next day - "2024-01-02T12:34junk", "...03Zjunk" -> trailing garbage ignored - "2024-01-02T12:34:" -> dangling component ignored - "2024-01-02T12:34:03+99:99" -> out-of-range tz, wrong date Root cause: the parser used unanchored sscanf calls that matched a prefix and ignored the rest, and it never range-checked the components. Replace it with a bounded cursor over the grammar YYYY<sep>MM<sep>DD [ (T|' '|D) HH:MM[:SS][.frac] [Z|(+|-)HH[:]?MM] ] (<sep> is '-' or '.', consistent within the date). The cursor must reach the end of the string, every field is a fixed digit width, and the date, time, and timezone components are range-checked; anything else is a domain error. All previously accepted valid forms — bare date, space/T/D separators, fractional seconds (any length), and Z / +HH:MM / -HHMM / offset suffixes — continue to round-trip. Extends the TIMESTAMP-cast coverage in ops/builtins_branch_cov.rfl with the separator, out-of-range, trailing-garbage, partial-component, and out-of-range-timezone rejection cases. * fix(builtins): validate timestamp cast bounds --------- Co-authored-by: Anton Kundenko <singaraiona@gmail.com> * fix(store): support nested column inserts and persistence (#365) * fix(eval): enforce restricted mode in compiled lambdas (#366) * feat(core): add bounded poll step for embedders (#367) * fix(test): rely on public runtime declarations (#369) * feat: website add consumer * feat(core): expose restricted poll mode (#372) * fix(aggr): preserve grouped nested first and last * fix(join): preserve nested columns (#376) * fix(csv): round-trip signed and >=24h TIME values in .csv.read (#379) * fix(str): propagate null start/length in substr instead of overflowing (#378) * fix(builtins): clamp out-of-range float in scalar numeric casts (#380) * fix(builtins): validate bounds in DATE string cast (#382) (as 'DATE str) parsed "YYYY.MM.DD" with an unanchored sscanf and never range-checked the month. A month greater than 13 walked the days-in-month table out of bounds — e.g. (as 'DATE "9999.99.99") reads md[99] on a 13-element array, which ASan reports as a heap/global out-of-bounds read (builtins.c:1567). The same path also silently accepted trailing garbage ("2024.01.02junk") and impossible days ("2024.02.31"). Replace it with a bounded cursor over YYYY.MM.DD that must consume the whole string, validates the month (1-12) BEFORE indexing the table, and validates the day against the actual number of days in that month (leap-aware). This mirrors the strict TIMESTAMP string cast. All valid dates — including the leap day 2024.02.29 — still round-trip; malformed input now returns a domain error instead of reading out of bounds or fabricating a date. Adds DATE-cast rejection/validation coverage to type/as.rfl. * perf(aggr): specialize reduction scans (#383) * fix(join): preserve nested columns Gather LIST columns with retained ownership across equi, anti, and asof joins. Represent unmatched boxed rows with the runtime null singleton and propagate allocation failures without dropping columns.\n\nFixes #375 * perf(aggr): specialize reduction scans * perf(collection): parallel radix distinct for fixed-width columns (#384) * perf(collection): parallel radix distinct for fixed-width columns distinct_vec_eager's hashset pass is single-threaded and dominates on large numeric/temporal columns. Reuse exec_count_distinct's radix layout (histogram -> scatter -> per-partition dedup) carrying row ids alongside values; first occurrences land in a shared byte array (a value lives in exactly one partition, so no atomics) and feed the existing sort + gather tail — result semantics unchanged. 13.9M-row column, 10 cores: 11.6K uniques 106->26ms, 125K uniques 164->45ms, 1.1M uniques 294->133ms, 5M uniques 719->502ms (the shared sequential value-sort now dominates that last case). * test(collection): cover the parallel radix distinct path The existing distinct.rfl only exercises small vectors; the radix kernel engages at >= 65536 rows. Add radix-scale assertions for every lane it handles (i64, i32, i16, f64, time, timestamp): value-sorted output, declared nulls collapsing to one sentinel, idempotence, the dedup invariant, agreement with the fused OP_COUNT_DISTINCT kernel, and the exact engage-threshold boundary. F64 NaN ordering among sorted values is comparator-defined, so those asserts check cardinality and null survival rather than full ordering. * fix(csv): harden import cancellation and schema handling (#387) * fix(csv): harden import cancellation and schemas * ci(tsan): use portable x86-64 baseline * fix(temporal): guard int64 overflow in DATE/TIMESTAMP decomposition (#386) * fix(temporal): guard int64 overflow in DATE/TIMESTAMP decomposition The DATE/TIMESTAMP → microseconds conversion shared by the temporal extract and truncate paths overflowed int64 on extreme inputs (UBSan): * DATE is int32 days, so an extreme value × µs-per-day overflowed — (dd (as 'DATE 2147483647)) tripped `raw * 86400000000`. * The ns→µs floor for TIMESTAMP negated the input, so a value within 999 of INT64_MIN overflowed `(-raw) + 999`. * The DAG date_trunc YEAR/MONTH arms re-multiply days_from_civil(...) — a day count floored down to the period start, up to a year beyond the input — by µs-per-day, so a DATE that cleared the µs bound still overflowed (d.year of (as 'DATE -106751991)). Both conversions appear in all four decomposition kernels: the standalone ray_temporal_extract / ray_temporal_truncate and the DAG exec_extract / exec_date_trunc morsel kernels. Do the TIMESTAMP ns→µs floor with truncate-then-adjust so it never negates (overflow-free, exact at INT64_MIN). A DATE so extreme its µs value is not representable decodes to a null instead of reading overflow garbage, consistent with how these kernels already treat a null input. The truncate DATE bound is the int64-NANOSECOND representable day range (not just the µs one) so the YEAR/MONTH re-multiply cannot overflow, and truncate additionally nulls a result whose bucketed µs would overflow the ns output. Adds temporal/extract_trunc_overflow.rfl covering the standalone kernels (yyyy/dd/mm/hh and (date …)) and the DAG kernels (dotted col.field), including the year/month re-multiply path, large-magnitude negative DATE, and minimum-edge TIMESTAMP. * fix(temporal): range-check truncated result, not pre-floor headroom rte_trunc_elem rejected a truncation whenever the input fell within one bucket of the low int64-ns edge, even when the floored result was still representable — so (date (as 'TIMESTAMP -9223286400000000000)) returned 0Np though its day boundary 1707.09.23 is a valid TIMESTAMP that the DAG path (select ts.date) returned, leaving the two public truncate paths disagreeing (PR #386 review). Floor with overflow-safe arithmetic (truncate toward zero, then guard the single toward-minus-infinity bucket subtraction) and range-check the actual bucketed result against the ns domain, mirroring exec_date_trunc. Adds the low-boundary case — standalone and DAG — as a regression. --------- Co-authored-by: Anton Kundenko <singaraiona@gmail.com> * fix(eval): materialize lazy values at compiled boundaries (#391) * fix(arith): wrap scalar integer add/sub/mul on overflow instead of UB (#388) ray_add_fn / ray_sub_fn / ray_mul_fn computed `as_i64(a) OP as_i64(b)` directly, so an i64 result that overflowed was signed-integer-overflow undefined behavior. UBSan reported it at src/ops/arith.c:141 (+), :223 (-), and :257 (*) for e.g. (+ 9223372036854775807 1), (- 0 -9223372036854775808), and (* 9223372036854775807 2). Compute the integer result with uint64 wraparound, matching the vector arithmetic kernel (src/ops/expr.c OP_ADD/SUB/MUL). The wrapped value — INT64_MIN for MAX+1, which is the i64 null sentinel — is exactly what the code already returned by relying on the overflow, so results are unchanged; only the undefined arithmetic is removed. Narrow-type (i16/i32) results are unaffected: their i64 intermediate never overflows and make_typed_int still narrows them. Adds test/rfl/arith/overflow_wrap.rfl. Co-authored-by: Anton Kundenko <singaraiona@gmail.com> * fix(builtins): reject malformed strings in TIME cast (#389) (as 'TIME str) parsed with an unanchored sscanf and never range-checked the minute/second fields, so it silently accepted malformed input and produced a wrong-but-valid value — the same data-corruption class the DATE (#382) and TIMESTAMP (#361) string casts were hardened against, but the TIME cast was left on the old path: - "12:34:56junk" -> 12:34:56.000 (trailing garbage ignored) - "25:99:99" -> 26:40:39.000 (minute/second out of range, normalized) Parse "[-]HH:MM[:SS][.fff]" with a bounded cursor that must consume the whole string, validating that minutes and seconds are 0-59. TIME is a signed duration — its ms-of-day may exceed a day and go negative (see .csv.read round-tripping) — so the hour field stays variable width and unbounded, and "HH:MM" without seconds plus a bare trailing "." are still accepted. The value is range-checked against the int32 millisecond domain. The string-vector cast routes through this same atom path, so it is fixed too. Extends the TIME-cast coverage in type/as.rfl with the duration, negative, and rejection cases. Co-authored-by: Anton Kundenko <singaraiona@gmail.com> --------- Co-authored-by: Karim <k.nassar@lynxtrading.com> Co-authored-by: Evgen <ebelozerov@lynxtrading.com> Co-authored-by: Evgen Byelozorov <belowzeroff@gmail.com> Co-authored-by: Evgen Belozerov <yevhenbielozorov@gmail.com> Co-authored-by: Serhii Savchuk <ser.vasilich@hotmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Problem
The DATE/TIMESTAMP → microseconds conversion shared by the temporal extract and truncate paths overflowed
int64on extreme inputs (UBSan):int32days, so an extreme value × µs-per-day overflows.-((-raw)+999)/1000), so a value within 999 ofINT64_MINoverflowed.Both appear in all four decomposition kernels: the standalone
ray_temporal_extract/ray_temporal_truncateand the DAGexec_extract/exec_date_truncmorsel kernels (reachable via(dd d)and via dotted-pathcol.yyyy/col.dateinselect).Fix
INT64_MIN.int64nanosecond output.Behaviour
(yyyy (as 'DATE 2147483647))0Nl(null)(mm (as 'TIMESTAMP -9223372036854775807))9(yyyy 2024.03.15)20242024(unchanged)select d.yyyyover[0 366 2147483647][2000 2001 0Nl]select d.dateover the same[… … 0Np]Ordinary values are unaffected. New coverage in
temporal/extract_trunc_overflow.rflexercises the standalone and both DAG kernels with extreme DATE, minimum-edge TIMESTAMP, and ordinary values. Fullmake testpasses (3660/3661, 1 skipped, 0 failed) under the default ASan/UBSan build.🤖 Generated with Claude Code